agora inbox for [email protected]help / color / mirror / Atom feed
[PATCH v10] Avoid orphaned objects dependencies 1275+ messages / 3 participants [nested] [flat]
* [PATCH v10] Avoid orphaned objects dependencies @ 2024-03-29 15:43 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Bertrand Drouvot @ 2024-03-29 15:43 UTC (permalink / raw) It's currently possible to create orphaned objects dependencies, for example: Scenario 1: session 1: begin; drop schema schem; session 2: create a function in the schema schem session 1: commit; With the above, the function created in session 2 would be linked to a non existing schema. Scenario 2: session 1: begin; create a function in the schema schem session 2: drop schema schem; session 1: commit; With the above, the function created in session 1 would be linked to a non existing schema. To avoid those scenarios, a new lock (that conflicts with a lock taken by DROP) has been put in place before the dependencies are being recorded. With this in place, the drop schema in scenario 2 would be locked. Also, after the new lock attempt, the patch checks that the object still exists: with this in place session 2 in scenario 1 would be locked and would report an error once session 1 committs (that would not be the case should session 1 abort the transaction). If the object is dropped before the new lock attempt is triggered then the patch would also report an error (but with less details). The patch adds a few tests for some dependency cases (that would currently produce orphaned objects): - schema and function (as the above scenarios) - alter a dependency (function and schema) - function and arg type - function and return type - function and function - domain and domain - table and type - server and foreign data wrapper --- src/backend/catalog/aclchk.c | 1 + src/backend/catalog/dependency.c | 109 ++++++++++++++- src/backend/catalog/heap.c | 8 ++ src/backend/catalog/index.c | 16 +++ src/backend/catalog/objectaddress.c | 57 ++++++++ src/backend/catalog/pg_aggregate.c | 9 ++ src/backend/catalog/pg_attrdef.c | 1 + src/backend/catalog/pg_cast.c | 5 + src/backend/catalog/pg_collation.c | 1 + src/backend/catalog/pg_constraint.c | 11 ++ src/backend/catalog/pg_conversion.c | 2 + src/backend/catalog/pg_depend.c | 27 +++- src/backend/catalog/pg_operator.c | 19 +++ src/backend/catalog/pg_proc.c | 17 ++- src/backend/catalog/pg_publication.c | 5 + src/backend/catalog/pg_range.c | 6 + src/backend/catalog/pg_type.c | 33 +++++ src/backend/catalog/toasting.c | 1 + src/backend/commands/alter.c | 4 + src/backend/commands/amcmds.c | 1 + src/backend/commands/cluster.c | 5 + src/backend/commands/event_trigger.c | 1 + src/backend/commands/extension.c | 5 + src/backend/commands/foreigncmds.c | 7 + src/backend/commands/functioncmds.c | 6 + src/backend/commands/indexcmds.c | 2 + src/backend/commands/opclasscmds.c | 18 +++ src/backend/commands/operatorcmds.c | 30 ++++ src/backend/commands/policy.c | 2 + src/backend/commands/proclang.c | 3 + src/backend/commands/sequence.c | 1 + src/backend/commands/statscmds.c | 3 + src/backend/commands/tablecmds.c | 32 +++-- src/backend/commands/trigger.c | 18 ++- src/backend/commands/tsearchcmds.c | 81 +++++++---- src/backend/commands/typecmds.c | 84 ++++++++++++ src/backend/rewrite/rewriteDefine.c | 1 + src/backend/utils/errcodes.txt | 1 + src/include/catalog/dependency.h | 7 + src/include/catalog/objectaddress.h | 1 + .../expected/test_dependencies_locks.out | 129 ++++++++++++++++++ src/test/isolation/isolation_schedule | 1 + .../specs/test_dependencies_locks.spec | 89 ++++++++++++ 43 files changed, 813 insertions(+), 47 deletions(-) 31.4% src/backend/catalog/ 36.4% src/backend/commands/ 18.5% src/test/isolation/expected/ 11.5% src/test/isolation/specs/ diff --git a/src/backend/catalog/aclchk.c b/src/backend/catalog/aclchk.c index 143876b77f..5a614f25ff 100644 --- a/src/backend/catalog/aclchk.c +++ b/src/backend/catalog/aclchk.c @@ -1413,6 +1413,7 @@ SetDefaultACL(InternalDefaultACL *iacls) referenced.objectId = iacls->nspid; referenced.objectSubId = 0; + LockNotPinnedObject(NamespaceRelationId, iacls->nspid); recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO); } } diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c index 0489cbabcb..abf8b92a6d 100644 --- a/src/backend/catalog/dependency.c +++ b/src/backend/catalog/dependency.c @@ -1519,6 +1519,84 @@ AcquireDeletionLock(const ObjectAddress *object, int flags) } } +/* + * LockNotPinnedObjectById + * + * Lock the object that we are about to record a dependency on. + * After it's locked, verify that it hasn't been dropped while we + * weren't looking. If the object has been dropped, this function + * does not return! + */ +void +LockNotPinnedObjectById(const ObjectAddress *object) +{ + char *object_description; + + if (isObjectPinned(object)) + return; + + /* + * Those don't rely on LockDatabaseObject() when being dropped (see + * AcquireDeletionLock()). Also it looks like they can not produce + * orphaned dependent objects when being dropped. + */ + if (object->classId == RelationRelationId || object->classId == AuthMemRelationId) + return; + + object_description = getObjectDescription(object, true); + + /* assume we should lock the whole object not a sub-object */ + LockDatabaseObject(object->classId, object->objectId, 0, AccessShareLock); + + /* check if object still exists */ + if (!ObjectByIdExist(object)) + { + if (object_description) + ereport(ERROR, + (errcode(ERRCODE_DEPENDENT_OBJECTS_DOES_NOT_EXIST), + errmsg("%s does not exist", object_description))); + else + ereport(ERROR, + (errcode(ERRCODE_DEPENDENT_OBJECTS_DOES_NOT_EXIST), + errmsg("a dependent object does not exist"))); + } + + if (object_description) + pfree(object_description); + + return; +} + +void +LockNotPinnedObjectsById(const ObjectAddress *object, int nobject) +{ + int i; + + if (nobject < 0) + return; + + for (i = 0; i < nobject; i++, object++) + LockNotPinnedObjectById(object); + + return; +} + + +/* + * LockNotPinnedObject + * + * Lock the object that we are about to record a dependency on. + */ +void +LockNotPinnedObject(Oid classid, Oid objid) +{ + ObjectAddress object; + + ObjectAddressSet(object, classid, objid); + + LockNotPinnedObjectById(&object); +} + /* * ReleaseDeletionLock - release an object deletion lock * @@ -1564,13 +1642,8 @@ recordDependencyOnExpr(const ObjectAddress *depender, /* Scan the expression tree for referenceable objects */ find_expr_references_walker(expr, &context); - /* Remove any duplicates */ - eliminate_duplicate_dependencies(context.addrs); - - /* And record 'em */ - recordMultipleDependencies(depender, - context.addrs->refs, context.addrs->numrefs, - behavior); + /* Record all of them (this includes duplicate elimination) */ + lock_record_object_address_dependencies(depender, context.addrs, behavior); free_object_addresses(context.addrs); } @@ -1654,14 +1727,19 @@ recordDependencyOnSingleRelExpr(const ObjectAddress *depender, /* Record the self-dependencies with the appropriate direction */ if (!reverse_self) + { + LockNotPinnedObjectsById(self_addrs->refs, self_addrs->numrefs); recordMultipleDependencies(depender, self_addrs->refs, self_addrs->numrefs, self_behavior); + } else { /* Can't use recordMultipleDependencies, so do it the hard way */ int selfref; + LockNotPinnedObjectById(depender); + for (selfref = 0; selfref < self_addrs->numrefs; selfref++) { ObjectAddress *thisobj = self_addrs->refs + selfref; @@ -1674,6 +1752,7 @@ recordDependencyOnSingleRelExpr(const ObjectAddress *depender, } /* Record the external dependencies */ + LockNotPinnedObjectsById(context.addrs->refs, context.addrs->numrefs); recordMultipleDependencies(depender, context.addrs->refs, context.addrs->numrefs, behavior); @@ -2734,6 +2813,22 @@ stack_address_present_add_flags(const ObjectAddress *object, return result; } +/* + * Record multiple dependencies from an ObjectAddresses array and lock the + * referenced objects, after first removing any duplicates. + */ +void +lock_record_object_address_dependencies(const ObjectAddress *depender, + ObjectAddresses *referenced, + DependencyType behavior) +{ + eliminate_duplicate_dependencies(referenced); + LockNotPinnedObjectsById(referenced->refs, referenced->numrefs); + recordMultipleDependencies(depender, + referenced->refs, referenced->numrefs, + behavior); +} + /* * Record multiple dependencies from an ObjectAddresses array, after first * removing any duplicates. diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index a122bbffce..72443710d3 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -843,6 +843,7 @@ AddNewAttributeTuples(Oid new_rel_oid, ObjectAddressSubSet(myself, RelationRelationId, new_rel_oid, i + 1); ObjectAddressSet(referenced, TypeRelationId, tupdesc->attrs[i].atttypid); + LockNotPinnedObject(TypeRelationId, tupdesc->attrs[i].atttypid); recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL); /* The default collation is pinned, so don't bother recording it */ @@ -851,6 +852,7 @@ AddNewAttributeTuples(Oid new_rel_oid, { ObjectAddressSet(referenced, CollationRelationId, tupdesc->attrs[i].attcollation); + LockNotPinnedObject(CollationRelationId, tupdesc->attrs[i].attcollation); recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL); } } @@ -1451,11 +1453,13 @@ heap_create_with_catalog(const char *relname, ObjectAddressSet(referenced, NamespaceRelationId, relnamespace); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(NamespaceRelationId, relnamespace); if (reloftypeid) { ObjectAddressSet(referenced, TypeRelationId, reloftypeid); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(TypeRelationId, reloftypeid); } /* @@ -1469,6 +1473,7 @@ heap_create_with_catalog(const char *relname, { ObjectAddressSet(referenced, AccessMethodRelationId, accessmtd); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(AccessMethodRelationId, accessmtd); } record_object_address_dependencies(&myself, addrs, DEPENDENCY_NORMAL); @@ -3383,6 +3388,7 @@ StorePartitionKey(Relation rel, { ObjectAddressSet(referenced, OperatorClassRelationId, partopclass[i]); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(OperatorClassRelationId, partopclass[i]); /* The default collation is pinned, so don't bother recording it */ if (OidIsValid(partcollation[i]) && @@ -3390,6 +3396,7 @@ StorePartitionKey(Relation rel, { ObjectAddressSet(referenced, CollationRelationId, partcollation[i]); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(CollationRelationId, partcollation[i]); } } @@ -3409,6 +3416,7 @@ StorePartitionKey(Relation rel, ObjectAddressSubSet(referenced, RelationRelationId, RelationGetRelid(rel), partattrs[i]); + /* Do we need a lock on RelationRelationId? */ recordDependencyOn(&referenced, &myself, DEPENDENCY_INTERNAL); } diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 55fdde4b24..b89c217a09 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1127,6 +1127,7 @@ index_create(Relation heapRelation, heapRelationId, indexInfo->ii_IndexAttrNumbers[i]); add_exact_object_address(&referenced, addrs); + /* XXX Do we need a lock for RelationRelationId? */ have_simple_col = true; } } @@ -1141,6 +1142,7 @@ index_create(Relation heapRelation, { ObjectAddressSet(referenced, RelationRelationId, heapRelationId); + /* XXX Do we need a lock for RelationRelationId? */ add_exact_object_address(&referenced, addrs); } @@ -1157,9 +1159,11 @@ index_create(Relation heapRelation, if (OidIsValid(parentIndexRelid)) { ObjectAddressSet(referenced, RelationRelationId, parentIndexRelid); + /* XXX Do we need a lock for RelationRelationId? */ recordDependencyOn(&myself, &referenced, DEPENDENCY_PARTITION_PRI); ObjectAddressSet(referenced, RelationRelationId, heapRelationId); + /* XXX Do we need a lock for RelationRelationId? */ recordDependencyOn(&myself, &referenced, DEPENDENCY_PARTITION_SEC); } @@ -1175,6 +1179,7 @@ index_create(Relation heapRelation, { ObjectAddressSet(referenced, CollationRelationId, collationIds[i]); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(CollationRelationId, collationIds[i]); } } @@ -1183,6 +1188,7 @@ index_create(Relation heapRelation, { ObjectAddressSet(referenced, OperatorClassRelationId, opclassIds[i]); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(OperatorClassRelationId, opclassIds[i]); } record_object_address_dependencies(&myself, addrs, DEPENDENCY_NORMAL); @@ -1987,6 +1993,14 @@ index_constraint_create(Relation heapRelation, */ ObjectAddressSet(myself, ConstraintRelationId, conOid); ObjectAddressSet(idxaddr, RelationRelationId, indexRelationId); + + /* + * CommandCounterIncrement here to ensure the new constraint entry is + * visible when we'll check of object existence when recording the + * dependencies. + */ + CommandCounterIncrement(); + LockNotPinnedObject(ConstraintRelationId, conOid); recordDependencyOn(&idxaddr, &myself, DEPENDENCY_INTERNAL); /* @@ -1998,9 +2012,11 @@ index_constraint_create(Relation heapRelation, ObjectAddress referenced; ObjectAddressSet(referenced, ConstraintRelationId, parentConstraintId); + LockNotPinnedObject(ConstraintRelationId, parentConstraintId); recordDependencyOn(&myself, &referenced, DEPENDENCY_PARTITION_PRI); ObjectAddressSet(referenced, RelationRelationId, RelationGetRelid(heapRelation)); + /* XXX Do we need a lock for RelationRelationId? */ recordDependencyOn(&myself, &referenced, DEPENDENCY_PARTITION_SEC); } diff --git a/src/backend/catalog/objectaddress.c b/src/backend/catalog/objectaddress.c index 7b536ac6fd..6d7abd3738 100644 --- a/src/backend/catalog/objectaddress.c +++ b/src/backend/catalog/objectaddress.c @@ -2590,6 +2590,63 @@ get_object_namespace(const ObjectAddress *address) return oid; } +/* + * ObjectByIdExist + * + * Return whether the given object exists. + * + * Works for most catalogs, if no special processing is needed. + */ +bool +ObjectByIdExist(const ObjectAddress *address) +{ + HeapTuple tuple; + int cache; + const ObjectPropertyType *property; + + property = get_object_property_data(address->classId); + + cache = property->oid_catcache_id; + + if (cache >= 0) + { + /* Fetch tuple from syscache. */ + tuple = SearchSysCache1(cache, ObjectIdGetDatum(address->objectId)); + + if (!HeapTupleIsValid(tuple)) + { + return false; + } + + ReleaseSysCache(tuple); + + return true; + } + else + { + Relation rel; + ScanKeyData skey[1]; + SysScanDesc scan; + + rel = table_open(address->classId, AccessShareLock); + + ScanKeyInit(&skey[0], + get_object_attnum_oid(address->classId), + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(address->objectId)); + + scan = systable_beginscan(rel, get_object_oid_index(address->classId), true, + NULL, 1, skey); + + /* we expect exactly one match */ + tuple = systable_getnext(scan); + systable_endscan(scan); + table_close(rel, AccessShareLock); + + return (HeapTupleIsValid(tuple)); + } +} + /* * Return ObjectType for the given object type as given by * getObjectTypeDescription; if no valid ObjectType code exists, but it's a diff --git a/src/backend/catalog/pg_aggregate.c b/src/backend/catalog/pg_aggregate.c index 90fc7db949..a47e3c5507 100644 --- a/src/backend/catalog/pg_aggregate.c +++ b/src/backend/catalog/pg_aggregate.c @@ -748,12 +748,14 @@ AggregateCreate(const char *aggName, /* Depends on transition function */ ObjectAddressSet(referenced, ProcedureRelationId, transfn); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(ProcedureRelationId, transfn); /* Depends on final function, if any */ if (OidIsValid(finalfn)) { ObjectAddressSet(referenced, ProcedureRelationId, finalfn); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(ProcedureRelationId, finalfn); } /* Depends on combine function, if any */ @@ -761,6 +763,7 @@ AggregateCreate(const char *aggName, { ObjectAddressSet(referenced, ProcedureRelationId, combinefn); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(ProcedureRelationId, combinefn); } /* Depends on serialization function, if any */ @@ -768,6 +771,7 @@ AggregateCreate(const char *aggName, { ObjectAddressSet(referenced, ProcedureRelationId, serialfn); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(ProcedureRelationId, serialfn); } /* Depends on deserialization function, if any */ @@ -775,6 +779,7 @@ AggregateCreate(const char *aggName, { ObjectAddressSet(referenced, ProcedureRelationId, deserialfn); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(ProcedureRelationId, deserialfn); } /* Depends on forward transition function, if any */ @@ -782,6 +787,7 @@ AggregateCreate(const char *aggName, { ObjectAddressSet(referenced, ProcedureRelationId, mtransfn); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(ProcedureRelationId, mtransfn); } /* Depends on inverse transition function, if any */ @@ -789,6 +795,7 @@ AggregateCreate(const char *aggName, { ObjectAddressSet(referenced, ProcedureRelationId, minvtransfn); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(ProcedureRelationId, minvtransfn); } /* Depends on final function, if any */ @@ -796,6 +803,7 @@ AggregateCreate(const char *aggName, { ObjectAddressSet(referenced, ProcedureRelationId, mfinalfn); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(ProcedureRelationId, mfinalfn); } /* Depends on sort operator, if any */ @@ -803,6 +811,7 @@ AggregateCreate(const char *aggName, { ObjectAddressSet(referenced, OperatorRelationId, sortop); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(OperatorRelationId, sortop); } record_object_address_dependencies(&myself, addrs, DEPENDENCY_NORMAL); diff --git a/src/backend/catalog/pg_attrdef.c b/src/backend/catalog/pg_attrdef.c index 003ae70b4d..80db2c0c20 100644 --- a/src/backend/catalog/pg_attrdef.c +++ b/src/backend/catalog/pg_attrdef.c @@ -178,6 +178,7 @@ StoreAttrDefault(Relation rel, AttrNumber attnum, colobject.objectId = RelationGetRelid(rel); colobject.objectSubId = attnum; + /* XXX Do we need a lock for RelationRelationId? */ recordDependencyOn(&defobject, &colobject, attgenerated ? DEPENDENCY_INTERNAL : DEPENDENCY_AUTO); diff --git a/src/backend/catalog/pg_cast.c b/src/backend/catalog/pg_cast.c index 5a5b855d51..d3707e424c 100644 --- a/src/backend/catalog/pg_cast.c +++ b/src/backend/catalog/pg_cast.c @@ -97,16 +97,19 @@ CastCreate(Oid sourcetypeid, Oid targettypeid, /* dependency on source type */ ObjectAddressSet(referenced, TypeRelationId, sourcetypeid); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(TypeRelationId, sourcetypeid); /* dependency on target type */ ObjectAddressSet(referenced, TypeRelationId, targettypeid); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(TypeRelationId, targettypeid); /* dependency on function */ if (OidIsValid(funcid)) { ObjectAddressSet(referenced, ProcedureRelationId, funcid); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(ProcedureRelationId, funcid); } /* dependencies on casts required for function */ @@ -114,11 +117,13 @@ CastCreate(Oid sourcetypeid, Oid targettypeid, { ObjectAddressSet(referenced, CastRelationId, incastid); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(CastRelationId, incastid); } if (OidIsValid(outcastid)) { ObjectAddressSet(referenced, CastRelationId, outcastid); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(CastRelationId, outcastid); } record_object_address_dependencies(&myself, addrs, behavior); diff --git a/src/backend/catalog/pg_collation.c b/src/backend/catalog/pg_collation.c index 7f2f701229..78498b8c20 100644 --- a/src/backend/catalog/pg_collation.c +++ b/src/backend/catalog/pg_collation.c @@ -218,6 +218,7 @@ CollationCreate(const char *collname, Oid collnamespace, referenced.classId = NamespaceRelationId; referenced.objectId = collnamespace; referenced.objectSubId = 0; + LockNotPinnedObject(NamespaceRelationId, collnamespace); recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL); /* create dependency on owner */ diff --git a/src/backend/catalog/pg_constraint.c b/src/backend/catalog/pg_constraint.c index 3baf9231ed..4afa9a1d69 100644 --- a/src/backend/catalog/pg_constraint.c +++ b/src/backend/catalog/pg_constraint.c @@ -257,12 +257,14 @@ CreateConstraintEntry(const char *constraintName, ObjectAddressSubSet(relobject, RelationRelationId, relId, constraintKey[i]); add_exact_object_address(&relobject, addrs_auto); + /* XXX Do we need a lock for RelationRelationId?? */ } } else { ObjectAddressSet(relobject, RelationRelationId, relId); add_exact_object_address(&relobject, addrs_auto); + /* XXX Do we need a lock for RelationRelationId?? */ } } @@ -275,6 +277,7 @@ CreateConstraintEntry(const char *constraintName, ObjectAddressSet(domobject, TypeRelationId, domainId); add_exact_object_address(&domobject, addrs_auto); + LockNotPinnedObject(TypeRelationId, domainId); } record_object_address_dependencies(&conobject, addrs_auto, @@ -299,12 +302,14 @@ CreateConstraintEntry(const char *constraintName, ObjectAddressSubSet(relobject, RelationRelationId, foreignRelId, foreignKey[i]); add_exact_object_address(&relobject, addrs_normal); + /* XXX Do we need a lock for RelationRelationId? */ } } else { ObjectAddressSet(relobject, RelationRelationId, foreignRelId); add_exact_object_address(&relobject, addrs_normal); + /* XXX Do we need a lock for RelationRelationId? */ } } @@ -320,6 +325,7 @@ CreateConstraintEntry(const char *constraintName, ObjectAddressSet(relobject, RelationRelationId, indexRelId); add_exact_object_address(&relobject, addrs_normal); + /* XXX Do we need a lock for RelationRelationId?? */ } if (foreignNKeys > 0) @@ -339,15 +345,18 @@ CreateConstraintEntry(const char *constraintName, { oprobject.objectId = pfEqOp[i]; add_exact_object_address(&oprobject, addrs_normal); + LockNotPinnedObject(OperatorRelationId, pfEqOp[i]); if (ppEqOp[i] != pfEqOp[i]) { oprobject.objectId = ppEqOp[i]; add_exact_object_address(&oprobject, addrs_normal); + LockNotPinnedObject(OperatorRelationId, ppEqOp[i]); } if (ffEqOp[i] != pfEqOp[i]) { oprobject.objectId = ffEqOp[i]; add_exact_object_address(&oprobject, addrs_normal); + LockNotPinnedObject(OperatorRelationId, ffEqOp[i]); } } } @@ -858,9 +867,11 @@ ConstraintSetParentConstraint(Oid childConstrId, ObjectAddressSet(depender, ConstraintRelationId, childConstrId); ObjectAddressSet(referenced, ConstraintRelationId, parentConstrId); + LockNotPinnedObject(ConstraintRelationId, parentConstrId); recordDependencyOn(&depender, &referenced, DEPENDENCY_PARTITION_PRI); ObjectAddressSet(referenced, RelationRelationId, childTableId); + /* XXX Do we need a lock for RelationRelationId? */ recordDependencyOn(&depender, &referenced, DEPENDENCY_PARTITION_SEC); } else diff --git a/src/backend/catalog/pg_conversion.c b/src/backend/catalog/pg_conversion.c index 0770878eac..25881654d6 100644 --- a/src/backend/catalog/pg_conversion.c +++ b/src/backend/catalog/pg_conversion.c @@ -116,12 +116,14 @@ ConversionCreate(const char *conname, Oid connamespace, referenced.classId = ProcedureRelationId; referenced.objectId = conproc; referenced.objectSubId = 0; + LockNotPinnedObject(ProcedureRelationId, conproc); recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL); /* create dependency on namespace */ referenced.classId = NamespaceRelationId; referenced.objectId = connamespace; referenced.objectSubId = 0; + LockNotPinnedObject(NamespaceRelationId, connamespace); recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL); /* create dependency on owner */ diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c index cfd7ef51df..b97aa2ab91 100644 --- a/src/backend/catalog/pg_depend.c +++ b/src/backend/catalog/pg_depend.c @@ -20,21 +20,20 @@ #include "catalog/catalog.h" #include "catalog/dependency.h" #include "catalog/indexing.h" +#include "catalog/pg_auth_members.h" #include "catalog/pg_constraint.h" #include "catalog/pg_depend.h" #include "catalog/pg_extension.h" #include "catalog/partition.h" #include "commands/extension.h" #include "miscadmin.h" +#include "storage/lock.h" #include "utils/fmgroids.h" #include "utils/lsyscache.h" #include "utils/syscache.h" #include "utils/rel.h" -static bool isObjectPinned(const ObjectAddress *object); - - /* * Record a dependency between 2 objects via their respective objectAddress. * The first argument is the dependent object, the second the one it @@ -100,6 +99,25 @@ recordMultipleDependencies(const ObjectAddress *depender, slot_init_count = 0; for (i = 0; i < nreferenced; i++, referenced++) { +#ifdef USE_ASSERT_CHECKING + LOCKTAG tag; + + SET_LOCKTAG_OBJECT(tag, + MyDatabaseId, + referenced->classId, + referenced->objectId, + 0); + + /* + * Assert the referenced object is locked (see + * LockNotPinnedObjectById()). + */ + Assert(isObjectPinned(referenced) || + referenced->classId == RelationRelationId || + referenced->classId == AuthMemRelationId || + LockHeldByMe(&tag, AccessShareLock)); +#endif + /* * If the referenced object is pinned by the system, there's no real * need to record dependencies on it. This saves lots of space in @@ -239,6 +257,7 @@ recordDependencyOnCurrentExtension(const ObjectAddress *object, extension.objectId = CurrentExtensionObject; extension.objectSubId = 0; + LockNotPinnedObject(ExtensionRelationId, CurrentExtensionObject); recordDependencyOn(object, &extension, DEPENDENCY_EXTENSION); } } @@ -706,7 +725,7 @@ changeDependenciesOn(Oid refClassId, Oid oldRefObjectId, * The passed subId, if any, is ignored; we assume that only whole objects * are pinned (and that this implies pinning their components). */ -static bool +bool isObjectPinned(const ObjectAddress *object) { return IsPinnedObject(object->classId, object->objectId); diff --git a/src/backend/catalog/pg_operator.c b/src/backend/catalog/pg_operator.c index 65b45a424a..e8374eec88 100644 --- a/src/backend/catalog/pg_operator.c +++ b/src/backend/catalog/pg_operator.c @@ -251,6 +251,16 @@ OperatorShellMake(const char *operatorName, values[Anum_pg_operator_oprrest - 1] = ObjectIdGetDatum(InvalidOid); values[Anum_pg_operator_oprjoin - 1] = ObjectIdGetDatum(InvalidOid); + /* Lock dependent objects */ + if (OidIsValid(operatorNamespace)) + LockNotPinnedObject(NamespaceRelationId, operatorNamespace); + + if (OidIsValid(leftTypeId)) + LockNotPinnedObject(TypeRelationId, leftTypeId); + + if (OidIsValid(rightTypeId)) + LockNotPinnedObject(TypeRelationId, rightTypeId); + /* * create a new operator tuple */ @@ -513,6 +523,15 @@ OperatorCreate(const char *operatorName, CatalogTupleInsert(pg_operator_desc, tup); } + /* Lock dependent objects */ + LockNotPinnedObject(NamespaceRelationId, operatorNamespace); + LockNotPinnedObject(TypeRelationId, leftTypeId); + LockNotPinnedObject(TypeRelationId, rightTypeId); + LockNotPinnedObject(TypeRelationId, operResultType); + LockNotPinnedObject(ProcedureRelationId, procedureId); + LockNotPinnedObject(ProcedureRelationId, restrictionId); + LockNotPinnedObject(ProcedureRelationId, joinId); + /* Add dependencies for the entry */ address = makeOperatorDependencies(tup, true, isUpdate); diff --git a/src/backend/catalog/pg_proc.c b/src/backend/catalog/pg_proc.c index 528c17cd7f..9cceb6413b 100644 --- a/src/backend/catalog/pg_proc.c +++ b/src/backend/catalog/pg_proc.c @@ -593,6 +593,13 @@ ProcedureCreate(const char *procedureName, if (is_update) deleteDependencyRecordsFor(ProcedureRelationId, retval, true); + /* + * CommandCounterIncrement here to ensure the new function entry is + * visible when we'll check of object existence when recording the + * dependencies. + */ + CommandCounterIncrement(); + addrs = new_object_addresses(); ObjectAddressSet(myself, ProcedureRelationId, retval); @@ -600,20 +607,24 @@ ProcedureCreate(const char *procedureName, /* dependency on namespace */ ObjectAddressSet(referenced, NamespaceRelationId, procNamespace); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(NamespaceRelationId, procNamespace); /* dependency on implementation language */ ObjectAddressSet(referenced, LanguageRelationId, languageObjectId); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(LanguageRelationId, languageObjectId); /* dependency on return type */ ObjectAddressSet(referenced, TypeRelationId, returnType); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(TypeRelationId, returnType); /* dependency on transform used by return type, if any */ if ((trfid = get_transform_oid(returnType, languageObjectId, true))) { ObjectAddressSet(referenced, TransformRelationId, trfid); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(TransformRelationId, trfid); } /* dependency on parameter types */ @@ -621,12 +632,14 @@ ProcedureCreate(const char *procedureName, { ObjectAddressSet(referenced, TypeRelationId, allParams[i]); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(TypeRelationId, allParams[i]); /* dependency on transform used by parameter type, if any */ if ((trfid = get_transform_oid(allParams[i], languageObjectId, true))) { ObjectAddressSet(referenced, TransformRelationId, trfid); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(TransformRelationId, trfid); } } @@ -635,6 +648,7 @@ ProcedureCreate(const char *procedureName, { ObjectAddressSet(referenced, ProcedureRelationId, prosupport); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(ProcedureRelationId, prosupport); } record_object_address_dependencies(&myself, addrs, DEPENDENCY_NORMAL); @@ -674,9 +688,6 @@ ProcedureCreate(const char *procedureName, ArrayType *set_items = NULL; int save_nestlevel = 0; - /* Advance command counter so new tuple can be seen by validator */ - CommandCounterIncrement(); - /* * Set per-function configuration parameters so that the validation is * done with the environment the function expects. However, if diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c index 0602398a54..5e16c0fcef 100644 --- a/src/backend/catalog/pg_publication.c +++ b/src/backend/catalog/pg_publication.c @@ -438,10 +438,12 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri, /* Add dependency on the publication */ ObjectAddressSet(referenced, PublicationRelationId, pubid); + LockNotPinnedObject(PublicationRelationId, pubid); recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO); /* Add dependency on the relation */ ObjectAddressSet(referenced, RelationRelationId, relid); + /* XXX Do we need a lock for RelationRelationId? */ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO); /* Add dependency on the objects mentioned in the qualifications */ @@ -454,6 +456,7 @@ publication_add_relation(Oid pubid, PublicationRelInfo *pri, for (int i = 0; i < natts; i++) { ObjectAddressSubSet(referenced, RelationRelationId, relid, attarray[i]); + /* XXX Do we need a lock for RelationRelationId? */ recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL); } @@ -661,10 +664,12 @@ publication_add_schema(Oid pubid, Oid schemaid, bool if_not_exists) /* Add dependency on the publication */ ObjectAddressSet(referenced, PublicationRelationId, pubid); + LockNotPinnedObject(PublicationRelationId, pubid); recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO); /* Add dependency on the schema */ ObjectAddressSet(referenced, NamespaceRelationId, schemaid); + LockNotPinnedObject(NamespaceRelationId, schemaid); recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO); /* Close the table */ diff --git a/src/backend/catalog/pg_range.c b/src/backend/catalog/pg_range.c index 501a6ba410..e5b5a0b6f8 100644 --- a/src/backend/catalog/pg_range.c +++ b/src/backend/catalog/pg_range.c @@ -70,26 +70,31 @@ RangeCreate(Oid rangeTypeOid, Oid rangeSubType, Oid rangeCollation, ObjectAddressSet(referenced, TypeRelationId, rangeSubType); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(TypeRelationId, rangeSubType); ObjectAddressSet(referenced, OperatorClassRelationId, rangeSubOpclass); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(OperatorClassRelationId, rangeSubOpclass); if (OidIsValid(rangeCollation)) { ObjectAddressSet(referenced, CollationRelationId, rangeCollation); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(CollationRelationId, rangeCollation); } if (OidIsValid(rangeCanonical)) { ObjectAddressSet(referenced, ProcedureRelationId, rangeCanonical); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(ProcedureRelationId, rangeCanonical); } if (OidIsValid(rangeSubDiff)) { ObjectAddressSet(referenced, ProcedureRelationId, rangeSubDiff); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(ProcedureRelationId, rangeSubDiff); } record_object_address_dependencies(&myself, addrs, DEPENDENCY_NORMAL); @@ -99,6 +104,7 @@ RangeCreate(Oid rangeTypeOid, Oid rangeSubType, Oid rangeCollation, referencing.classId = TypeRelationId; referencing.objectId = multirangeTypeOid; referencing.objectSubId = 0; + LockNotPinnedObject(TypeRelationId, rangeTypeOid); recordDependencyOn(&referencing, &myself, DEPENDENCY_INTERNAL); table_close(pg_range, RowExclusiveLock); diff --git a/src/backend/catalog/pg_type.c b/src/backend/catalog/pg_type.c index 395dec8ed8..92614cdaaa 100644 --- a/src/backend/catalog/pg_type.c +++ b/src/backend/catalog/pg_type.c @@ -157,6 +157,12 @@ TypeShellMake(const char *typeName, Oid typeNamespace, Oid ownerId) * Create dependencies. We can/must skip this in bootstrap mode. */ if (!IsBootstrapProcessingMode()) + { + /* Lock dependent objects */ + LockNotPinnedObject(NamespaceRelationId, typeNamespace); + LockNotPinnedObject(ProcedureRelationId, F_SHELL_IN); + LockNotPinnedObject(ProcedureRelationId, F_SHELL_OUT); + GenerateTypeDependencies(tup, pg_type_desc, NULL, @@ -166,6 +172,7 @@ TypeShellMake(const char *typeName, Oid typeNamespace, Oid ownerId) false, true, /* make extension dependency */ false); + } /* Post creation hook for new shell type */ InvokeObjectPostCreateHook(TypeRelationId, typoid, 0); @@ -494,6 +501,31 @@ TypeCreate(Oid newTypeOid, * Create dependencies. We can/must skip this in bootstrap mode. */ if (!IsBootstrapProcessingMode()) + { + /* + * CommandCounterIncrement here to ensure the new type entry is + * visible when we'll check of object existence when recording the + * dependencies. + */ + CommandCounterIncrement(); + + /* Lock dependent objects */ + LockNotPinnedObject(NamespaceRelationId, typeNamespace); + LockNotPinnedObject(ProcedureRelationId, inputProcedure); + LockNotPinnedObject(ProcedureRelationId, outputProcedure); + LockNotPinnedObject(ProcedureRelationId, receiveProcedure); + LockNotPinnedObject(ProcedureRelationId, sendProcedure); + LockNotPinnedObject(ProcedureRelationId, typmodinProcedure); + LockNotPinnedObject(ProcedureRelationId, typmodoutProcedure); + LockNotPinnedObject(ProcedureRelationId, analyzeProcedure); + LockNotPinnedObject(ProcedureRelationId, subscriptProcedure); + LockNotPinnedObject(TypeRelationId, baseType); + LockNotPinnedObject(CollationRelationId, typeCollation); + LockNotPinnedObject(TypeRelationId, elementType); + /* XXX Do we need a lock for RelationRelationId? */ + if (relationKind == RELKIND_COMPOSITE_TYPE) + LockNotPinnedObject(TypeRelationId, typeObjectId); + GenerateTypeDependencies(tup, pg_type_desc, (defaultTypeBin ? @@ -505,6 +537,7 @@ TypeCreate(Oid newTypeOid, isDependentType, true, /* make extension dependency */ rebuildDeps); + } /* Post creation hook for new type */ InvokeObjectPostCreateHook(TypeRelationId, typeObjectId, 0); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 738bc46ae8..4a708030ac 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -367,6 +367,7 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, toastobject.objectId = toast_relid; toastobject.objectSubId = 0; + /* XXX Do we need a lock for RelationRelationId? */ recordDependencyOn(&toastobject, &baseobject, DEPENDENCY_INTERNAL); } diff --git a/src/backend/commands/alter.c b/src/backend/commands/alter.c index 4f99ebb447..57e86f576a 100644 --- a/src/backend/commands/alter.c +++ b/src/backend/commands/alter.c @@ -501,7 +501,10 @@ ExecAlterObjectDependsStmt(AlterObjectDependsStmt *stmt, ObjectAddress *refAddre currexts = getAutoExtensionsOfObject(address.classId, address.objectId); if (!list_member_oid(currexts, refAddr.objectId)) + { + LockNotPinnedObject(refAddr.classId, refAddr.objectId); recordDependencyOn(&address, &refAddr, DEPENDENCY_AUTO_EXTENSION); + } } return address; @@ -807,6 +810,7 @@ AlterObjectNamespace_internal(Relation rel, Oid objid, Oid nspOid) pfree(replaces); /* update dependency to point to the new schema */ + LockNotPinnedObject(NamespaceRelationId, nspOid); if (changeDependencyFor(classId, objid, NamespaceRelationId, oldNspOid, nspOid) != 1) elog(ERROR, "could not change schema dependency for object %u", diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index aaa0f9a1dc..8616a7c9fa 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -104,6 +104,7 @@ CreateAccessMethod(CreateAmStmt *stmt) referenced.objectId = amhandler; referenced.objectSubId = 0; + LockNotPinnedObject(ProcedureRelationId, amhandler); recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL); recordDependencyOnCurrentExtension(&myself, false); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 78f96789b0..b54a04f4b0 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1272,6 +1272,7 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class, */ if (relam1 != relam2) { + LockNotPinnedObject(AccessMethodRelationId, relam2); if (changeDependencyFor(RelationRelationId, r1, AccessMethodRelationId, @@ -1280,6 +1281,8 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class, elog(ERROR, "could not change access method dependency for relation \"%s.%s\"", get_namespace_name(get_rel_namespace(r1)), get_rel_name(r1)); + + LockNotPinnedObject(AccessMethodRelationId, relam1); if (changeDependencyFor(RelationRelationId, r2, AccessMethodRelationId, @@ -1381,6 +1384,7 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class, { baseobject.objectId = r1; toastobject.objectId = relform1->reltoastrelid; + /* XXX Do we need a lock for RelationRelationId? */ recordDependencyOn(&toastobject, &baseobject, DEPENDENCY_INTERNAL); } @@ -1389,6 +1393,7 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class, { baseobject.objectId = r2; toastobject.objectId = relform2->reltoastrelid; + /* XXX Do we need a lock for RelationRelationId? */ recordDependencyOn(&toastobject, &baseobject, DEPENDENCY_INTERNAL); } diff --git a/src/backend/commands/event_trigger.c b/src/backend/commands/event_trigger.c index 7a5ed6b985..8d0cdec59e 100644 --- a/src/backend/commands/event_trigger.c +++ b/src/backend/commands/event_trigger.c @@ -327,6 +327,7 @@ insert_event_trigger_tuple(const char *trigname, const char *eventname, Oid evtO referenced.classId = ProcedureRelationId; referenced.objectId = funcoid; referenced.objectSubId = 0; + LockNotPinnedObject(ProcedureRelationId, funcoid); recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL); /* Depend on extension, if any. */ diff --git a/src/backend/commands/extension.c b/src/backend/commands/extension.c index 1643c8c69a..669a5d6dd8 100644 --- a/src/backend/commands/extension.c +++ b/src/backend/commands/extension.c @@ -1924,6 +1924,7 @@ InsertExtensionTuple(const char *extName, Oid extOwner, ObjectAddressSet(nsp, NamespaceRelationId, schemaOid); add_exact_object_address(&nsp, refobjs); + LockNotPinnedObject(NamespaceRelationId, schemaOid); foreach(lc, requiredExtensions) { @@ -1932,6 +1933,7 @@ InsertExtensionTuple(const char *extName, Oid extOwner, ObjectAddressSet(otherext, ExtensionRelationId, reqext); add_exact_object_address(&otherext, refobjs); + LockNotPinnedObject(ExtensionRelationId, reqext); } /* Record all of them (this includes duplicate elimination) */ @@ -2968,6 +2970,7 @@ AlterExtensionNamespace(const char *extensionName, const char *newschema, Oid *o table_close(extRel, RowExclusiveLock); /* update dependency to point to the new schema */ + LockNotPinnedObject(NamespaceRelationId, nspOid); if (changeDependencyFor(ExtensionRelationId, extensionOid, NamespaceRelationId, oldNspOid, nspOid) != 1) elog(ERROR, "could not change schema dependency for extension %s", @@ -3258,6 +3261,7 @@ ApplyExtensionUpdates(Oid extensionOid, otherext.objectId = reqext; otherext.objectSubId = 0; + LockNotPinnedObject(ExtensionRelationId, reqext); recordDependencyOn(&myself, &otherext, DEPENDENCY_NORMAL); } @@ -3414,6 +3418,7 @@ ExecAlterExtensionContentsRecurse(AlterExtensionContentsStmt *stmt, /* * OK, add the dependency. */ + LockNotPinnedObject(extension.classId, extension.objectId); recordDependencyOn(&object, &extension, DEPENDENCY_EXTENSION); /* diff --git a/src/backend/commands/foreigncmds.c b/src/backend/commands/foreigncmds.c index cf61bbac1f..735bca486c 100644 --- a/src/backend/commands/foreigncmds.c +++ b/src/backend/commands/foreigncmds.c @@ -642,6 +642,7 @@ CreateForeignDataWrapper(ParseState *pstate, CreateFdwStmt *stmt) referenced.classId = ProcedureRelationId; referenced.objectId = fdwhandler; referenced.objectSubId = 0; + LockNotPinnedObject(ProcedureRelationId, fdwhandler); recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL); } @@ -650,6 +651,7 @@ CreateForeignDataWrapper(ParseState *pstate, CreateFdwStmt *stmt) referenced.classId = ProcedureRelationId; referenced.objectId = fdwvalidator; referenced.objectSubId = 0; + LockNotPinnedObject(ProcedureRelationId, fdwvalidator); recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL); } @@ -811,6 +813,7 @@ AlterForeignDataWrapper(ParseState *pstate, AlterFdwStmt *stmt) referenced.classId = ProcedureRelationId; referenced.objectId = fdwhandler; referenced.objectSubId = 0; + LockNotPinnedObject(ProcedureRelationId, fdwhandler); recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL); } @@ -819,6 +822,7 @@ AlterForeignDataWrapper(ParseState *pstate, AlterFdwStmt *stmt) referenced.classId = ProcedureRelationId; referenced.objectId = fdwvalidator; referenced.objectSubId = 0; + LockNotPinnedObject(ProcedureRelationId, fdwvalidator); recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL); } } @@ -951,6 +955,7 @@ CreateForeignServer(CreateForeignServerStmt *stmt) referenced.classId = ForeignDataWrapperRelationId; referenced.objectId = fdw->fdwid; referenced.objectSubId = 0; + LockNotPinnedObject(ForeignDataWrapperRelationId, fdw->fdwid); recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL); recordDependencyOnOwner(ForeignServerRelationId, srvId, ownerId); @@ -1195,6 +1200,7 @@ CreateUserMapping(CreateUserMappingStmt *stmt) referenced.classId = ForeignServerRelationId; referenced.objectId = srv->serverid; referenced.objectSubId = 0; + LockNotPinnedObject(ForeignServerRelationId, srv->serverid); recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL); if (OidIsValid(useId)) @@ -1472,6 +1478,7 @@ CreateForeignTable(CreateForeignTableStmt *stmt, Oid relid) referenced.classId = ForeignServerRelationId; referenced.objectId = server->serverid; referenced.objectSubId = 0; + LockNotPinnedObject(ForeignServerRelationId, server->serverid); recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL); table_close(ftrel, RowExclusiveLock); diff --git a/src/backend/commands/functioncmds.c b/src/backend/commands/functioncmds.c index 6593fd7d81..8207ef08b3 100644 --- a/src/backend/commands/functioncmds.c +++ b/src/backend/commands/functioncmds.c @@ -1446,6 +1446,7 @@ AlterFunction(ParseState *pstate, AlterFunctionStmt *stmt) /* Add or replace dependency on support function */ if (OidIsValid(procForm->prosupport)) { + LockNotPinnedObject(ProcedureRelationId, newsupport); if (changeDependencyFor(ProcedureRelationId, funcOid, ProcedureRelationId, procForm->prosupport, newsupport) != 1) @@ -1459,6 +1460,7 @@ AlterFunction(ParseState *pstate, AlterFunctionStmt *stmt) referenced.classId = ProcedureRelationId; referenced.objectId = newsupport; referenced.objectSubId = 0; + LockNotPinnedObject(ProcedureRelationId, newsupport); recordDependencyOn(&address, &referenced, DEPENDENCY_NORMAL); } @@ -1962,21 +1964,25 @@ CreateTransform(CreateTransformStmt *stmt) /* dependency on language */ ObjectAddressSet(referenced, LanguageRelationId, langid); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(LanguageRelationId, langid); /* dependency on type */ ObjectAddressSet(referenced, TypeRelationId, typeid); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(TypeRelationId, typeid); /* dependencies on functions */ if (OidIsValid(fromsqlfuncid)) { ObjectAddressSet(referenced, ProcedureRelationId, fromsqlfuncid); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(ProcedureRelationId, fromsqlfuncid); } if (OidIsValid(tosqlfuncid)) { ObjectAddressSet(referenced, ProcedureRelationId, tosqlfuncid); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(ProcedureRelationId, tosqlfuncid); } record_object_address_dependencies(&myself, addrs, DEPENDENCY_NORMAL); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 309389e20d..b14eaad7a3 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -4377,8 +4377,10 @@ IndexSetParentIndex(Relation partitionIdx, Oid parentOid) ObjectAddressSet(parentIdx, RelationRelationId, parentOid); ObjectAddressSet(partitionTbl, RelationRelationId, partitionIdx->rd_index->indrelid); + /* Do we lock for RelationRelationId?? */ recordDependencyOn(&partIdx, &parentIdx, DEPENDENCY_PARTITION_PRI); + /* Do we lock for RelationRelationId?? */ recordDependencyOn(&partIdx, &partitionTbl, DEPENDENCY_PARTITION_SEC); } diff --git a/src/backend/commands/opclasscmds.c b/src/backend/commands/opclasscmds.c index b8b5c147c5..ca15106ca9 100644 --- a/src/backend/commands/opclasscmds.c +++ b/src/backend/commands/opclasscmds.c @@ -298,12 +298,14 @@ CreateOpFamily(CreateOpFamilyStmt *stmt, const char *opfname, referenced.classId = AccessMethodRelationId; referenced.objectId = amoid; referenced.objectSubId = 0; + LockNotPinnedObject(AccessMethodRelationId, amoid); recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO); /* dependency on namespace */ referenced.classId = NamespaceRelationId; referenced.objectId = namespaceoid; referenced.objectSubId = 0; + LockNotPinnedObject(NamespaceRelationId, namespaceoid); recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL); /* dependency on owner */ @@ -725,18 +727,21 @@ DefineOpClass(CreateOpClassStmt *stmt) referenced.classId = NamespaceRelationId; referenced.objectId = namespaceoid; referenced.objectSubId = 0; + LockNotPinnedObject(NamespaceRelationId, namespaceoid); recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL); /* dependency on opfamily */ referenced.classId = OperatorFamilyRelationId; referenced.objectId = opfamilyoid; referenced.objectSubId = 0; + LockNotPinnedObject(OperatorFamilyRelationId, opfamilyoid); recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO); /* dependency on indexed datatype */ referenced.classId = TypeRelationId; referenced.objectId = typeoid; referenced.objectSubId = 0; + LockNotPinnedObject(TypeRelationId, typeoid); recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL); /* dependency on storage datatype */ @@ -745,6 +750,7 @@ DefineOpClass(CreateOpClassStmt *stmt) referenced.classId = TypeRelationId; referenced.objectId = storageoid; referenced.objectSubId = 0; + LockNotPinnedObject(TypeRelationId, storageoid); recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL); } @@ -1486,6 +1492,13 @@ storeOperators(List *opfamilyname, Oid amoid, Oid opfamilyoid, heap_freetuple(tup); + /* + * CommandCounterIncrement here to ensure the new operator entry is + * visible when we'll check of object existence when recording the + * dependencies. + */ + CommandCounterIncrement(); + /* Make its dependencies */ myself.classId = AccessMethodOperatorRelationId; myself.objectId = entryoid; @@ -1496,6 +1509,7 @@ storeOperators(List *opfamilyname, Oid amoid, Oid opfamilyoid, referenced.objectSubId = 0; /* see comments in amapi.h about dependency strength */ + LockNotPinnedObject(OperatorRelationId, op->object); recordDependencyOn(&myself, &referenced, op->ref_is_hard ? DEPENDENCY_NORMAL : DEPENDENCY_AUTO); @@ -1504,6 +1518,7 @@ storeOperators(List *opfamilyname, Oid amoid, Oid opfamilyoid, referenced.objectId = op->refobjid; referenced.objectSubId = 0; + LockNotPinnedObject(referenced.classId, op->refobjid); recordDependencyOn(&myself, &referenced, op->ref_is_hard ? DEPENDENCY_INTERNAL : DEPENDENCY_AUTO); @@ -1514,6 +1529,7 @@ storeOperators(List *opfamilyname, Oid amoid, Oid opfamilyoid, referenced.objectId = op->sortfamily; referenced.objectSubId = 0; + LockNotPinnedObject(OperatorFamilyRelationId, op->sortfamily); recordDependencyOn(&myself, &referenced, op->ref_is_hard ? DEPENDENCY_NORMAL : DEPENDENCY_AUTO); } @@ -1597,6 +1613,7 @@ storeProcedures(List *opfamilyname, Oid amoid, Oid opfamilyoid, referenced.objectSubId = 0; /* see comments in amapi.h about dependency strength */ + LockNotPinnedObject(ProcedureRelationId, proc->object); recordDependencyOn(&myself, &referenced, proc->ref_is_hard ? DEPENDENCY_NORMAL : DEPENDENCY_AUTO); @@ -1605,6 +1622,7 @@ storeProcedures(List *opfamilyname, Oid amoid, Oid opfamilyoid, referenced.objectId = proc->refobjid; referenced.objectSubId = 0; + LockNotPinnedObject(referenced.classId, proc->refobjid); recordDependencyOn(&myself, &referenced, proc->ref_is_hard ? DEPENDENCY_INTERNAL : DEPENDENCY_AUTO); diff --git a/src/backend/commands/operatorcmds.c b/src/backend/commands/operatorcmds.c index 5872a3e192..58a69e7cc2 100644 --- a/src/backend/commands/operatorcmds.c +++ b/src/backend/commands/operatorcmds.c @@ -33,6 +33,7 @@ #include "access/htup_details.h" #include "access/table.h" +#include "catalog/dependency.h" #include "catalog/indexing.h" #include "catalog/objectaccess.h" #include "catalog/pg_namespace.h" @@ -656,11 +657,15 @@ AlterOperator(AlterOperatorStmt *stmt) { replaces[Anum_pg_operator_oprrest - 1] = true; values[Anum_pg_operator_oprrest - 1] = ObjectIdGetDatum(restrictionOid); + if (OidIsValid(restrictionOid)) + LockNotPinnedObject(ProcedureRelationId, restrictionOid); } if (updateJoin) { replaces[Anum_pg_operator_oprjoin - 1] = true; values[Anum_pg_operator_oprjoin - 1] = ObjectIdGetDatum(joinOid); + if (OidIsValid(joinOid)) + LockNotPinnedObject(ProcedureRelationId, joinOid); } if (OidIsValid(commutatorOid)) { @@ -688,6 +693,31 @@ AlterOperator(AlterOperatorStmt *stmt) CatalogTupleUpdate(catalog, &tup->t_self, tup); + + /* Lock dependent objects */ + oprForm = (Form_pg_operator) GETSTRUCT(tup); + + if (OidIsValid(oprForm->oprnamespace)) + LockNotPinnedObject(NamespaceRelationId, oprForm->oprnamespace); + + if (OidIsValid(oprForm->oprleft)) + LockNotPinnedObject(TypeRelationId, oprForm->oprleft); + + if (OidIsValid(oprForm->oprright)) + LockNotPinnedObject(TypeRelationId, oprForm->oprright); + + if (OidIsValid(oprForm->oprresult)) + LockNotPinnedObject(TypeRelationId, oprForm->oprresult); + + if (OidIsValid(oprForm->oprcode)) + LockNotPinnedObject(ProcedureRelationId, oprForm->oprcode); + + if (OidIsValid(oprForm->oprrest)) + LockNotPinnedObject(ProcedureRelationId, oprForm->oprrest); + + if (OidIsValid(oprForm->oprjoin)) + LockNotPinnedObject(ProcedureRelationId, oprForm->oprjoin); + address = makeOperatorDependencies(tup, false, true); if (OidIsValid(commutatorOid) || OidIsValid(negatorOid)) diff --git a/src/backend/commands/policy.c b/src/backend/commands/policy.c index 6ff3eba824..31e61e4e67 100644 --- a/src/backend/commands/policy.c +++ b/src/backend/commands/policy.c @@ -722,6 +722,7 @@ CreatePolicy(CreatePolicyStmt *stmt) myself.objectId = policy_id; myself.objectSubId = 0; + /* XXX Do we need a lock for RelationRelationId? */ recordDependencyOn(&myself, &target, DEPENDENCY_AUTO); recordDependencyOnExpr(&myself, qual, qual_pstate->p_rtable, @@ -1053,6 +1054,7 @@ AlterPolicy(AlterPolicyStmt *stmt) myself.objectId = policy_id; myself.objectSubId = 0; + /* XXX Do we need a lock for RelationRelationId? */ recordDependencyOn(&myself, &target, DEPENDENCY_AUTO); recordDependencyOnExpr(&myself, qual, qual_parse_rtable, DEPENDENCY_NORMAL); diff --git a/src/backend/commands/proclang.c b/src/backend/commands/proclang.c index 881f90017e..fadfd9064f 100644 --- a/src/backend/commands/proclang.c +++ b/src/backend/commands/proclang.c @@ -190,12 +190,14 @@ CreateProceduralLanguage(CreatePLangStmt *stmt) /* dependency on the PL handler function */ ObjectAddressSet(referenced, ProcedureRelationId, handlerOid); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(ProcedureRelationId, handlerOid); /* dependency on the inline handler function, if any */ if (OidIsValid(inlineOid)) { ObjectAddressSet(referenced, ProcedureRelationId, inlineOid); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(ProcedureRelationId, inlineOid); } /* dependency on the validator function, if any */ @@ -203,6 +205,7 @@ CreateProceduralLanguage(CreatePLangStmt *stmt) { ObjectAddressSet(referenced, ProcedureRelationId, valOid); add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(ProcedureRelationId, valOid); } record_object_address_dependencies(&myself, addrs, DEPENDENCY_NORMAL); diff --git a/src/backend/commands/sequence.c b/src/backend/commands/sequence.c index 28f8522264..b1db4e8933 100644 --- a/src/backend/commands/sequence.c +++ b/src/backend/commands/sequence.c @@ -1681,6 +1681,7 @@ process_owned_by(Relation seqrel, List *owned_by, bool for_identity) depobject.classId = RelationRelationId; depobject.objectId = RelationGetRelid(seqrel); depobject.objectSubId = 0; + /* XXX Do we need a lock for RelationRelationId? */ recordDependencyOn(&depobject, &refobject, deptype); } diff --git a/src/backend/commands/statscmds.c b/src/backend/commands/statscmds.c index 1db3ef69d2..372ae02650 100644 --- a/src/backend/commands/statscmds.c +++ b/src/backend/commands/statscmds.c @@ -536,6 +536,7 @@ CreateStatistics(CreateStatsStmt *stmt) for (i = 0; i < nattnums; i++) { ObjectAddressSubSet(parentobject, RelationRelationId, relid, attnums[i]); + /* XXX Do we need a lock for RelationRelationId? */ recordDependencyOn(&myself, &parentobject, DEPENDENCY_AUTO); } @@ -553,6 +554,7 @@ CreateStatistics(CreateStatsStmt *stmt) if (!nattnums) { ObjectAddressSet(parentobject, RelationRelationId, relid); + /* XXX Do we need a lock for RelationRelationId? */ recordDependencyOn(&myself, &parentobject, DEPENDENCY_AUTO); } @@ -573,6 +575,7 @@ CreateStatistics(CreateStatsStmt *stmt) * than the underlying table(s). */ ObjectAddressSet(parentobject, NamespaceRelationId, namespaceId); + LockNotPinnedObject(NamespaceRelationId, namespaceId); recordDependencyOn(&myself, &parentobject, DEPENDENCY_NORMAL); recordDependencyOnOwner(StatisticExtRelationId, statoid, stxowner); diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 66cda26a25..c66f110984 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -3438,6 +3438,7 @@ StoreCatalogInheritance1(Oid relationId, Oid parentOid, childobject.objectId = relationId; childobject.objectSubId = 0; + /* XXX Do we need a lock for RelationRelationId? */ recordDependencyOn(&childobject, &parentobject, child_dependency_type(child_is_partition)); @@ -7349,7 +7350,9 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, /* * Add needed dependency entries for the new column. */ + LockNotPinnedObject(TypeRelationId, attribute->atttypid); add_column_datatype_dependency(myrelid, newattnum, attribute->atttypid); + LockNotPinnedObject(CollationRelationId, attribute->attcollation); add_column_collation_dependency(myrelid, newattnum, attribute->attcollation); /* @@ -10174,6 +10177,7 @@ addFkRecurseReferenced(List **wqueue, Constraint *fkconstraint, Relation rel, ObjectAddress referenced; ObjectAddressSet(referenced, ConstraintRelationId, parentConstr); + LockNotPinnedObject(ConstraintRelationId, parentConstr); recordDependencyOn(&address, &referenced, DEPENDENCY_INTERNAL); } @@ -10465,8 +10469,10 @@ addFkRecurseReferencing(List **wqueue, Constraint *fkconstraint, Relation rel, */ ObjectAddressSet(address, ConstraintRelationId, constrOid); ObjectAddressSet(referenced, ConstraintRelationId, parentConstr); + LockNotPinnedObject(ConstraintRelationId, parentConstr); recordDependencyOn(&address, &referenced, DEPENDENCY_PARTITION_PRI); ObjectAddressSet(referenced, RelationRelationId, partitionId); + /* XXX Do we need a lock for RelationRelationId? */ recordDependencyOn(&address, &referenced, DEPENDENCY_PARTITION_SEC); /* Make all this visible before recursing */ @@ -10967,9 +10973,11 @@ CloneFkReferencing(List **wqueue, Relation parentRel, Relation partRel) /* Set up partition dependencies for the new constraint */ ObjectAddressSet(address, ConstraintRelationId, constrOid); ObjectAddressSet(referenced, ConstraintRelationId, parentConstrOid); + LockDatabaseObject(ConstraintRelationId, parentConstrOid, 0, AccessShareLock); recordDependencyOn(&address, &referenced, DEPENDENCY_PARTITION_PRI); ObjectAddressSet(referenced, RelationRelationId, RelationGetRelid(partRel)); + /* XXX Do we need a lock for RelationRelationId? */ recordDependencyOn(&address, &referenced, DEPENDENCY_PARTITION_SEC); /* Done with the cloned constraint's tuple */ @@ -13254,7 +13262,9 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, table_close(attrelation, RowExclusiveLock); /* Install dependencies on new datatype and collation */ + LockNotPinnedObject(TypeRelationId, targettype); add_column_datatype_dependency(RelationGetRelid(rel), attnum, targettype); + LockNotPinnedObject(CollationRelationId, targetcollid); add_column_collation_dependency(RelationGetRelid(rel), attnum, targetcollid); /* @@ -14816,6 +14826,7 @@ ATExecSetAccessMethodNoStorage(Relation rel, Oid newAccessMethodId) */ ObjectAddressSet(relobj, RelationRelationId, reloid); ObjectAddressSet(referenced, AccessMethodRelationId, rd_rel->relam); + LockNotPinnedObject(AccessMethodRelationId, rd_rel->relam); recordDependencyOn(&relobj, &referenced, DEPENDENCY_NORMAL); } else if (OidIsValid(oldAccessMethodId) && @@ -14835,6 +14846,7 @@ ATExecSetAccessMethodNoStorage(Relation rel, Oid newAccessMethodId) OidIsValid(rd_rel->relam)); /* Both are valid, so update the dependency */ + LockNotPinnedObject(AccessMethodRelationId, rd_rel->relam); changeDependencyFor(RelationRelationId, reloid, AccessMethodRelationId, oldAccessMethodId, rd_rel->relam); @@ -16434,6 +16446,7 @@ ATExecAddOf(Relation rel, const TypeName *ofTypename, LOCKMODE lockmode) typeobj.classId = TypeRelationId; typeobj.objectId = typeid; typeobj.objectSubId = 0; + LockNotPinnedObject(TypeRelationId, typeid); recordDependencyOn(&tableobj, &typeobj, DEPENDENCY_NORMAL); /* Update pg_class.reloftype */ @@ -17192,14 +17205,17 @@ AlterRelationNamespaceInternal(Relation classRel, Oid relOid, CatalogTupleUpdate(classRel, &classTup->t_self, classTup); /* Update dependency on schema if caller said so */ - if (hasDependEntry && - changeDependencyFor(RelationRelationId, - relOid, - NamespaceRelationId, - oldNspOid, - newNspOid) != 1) - elog(ERROR, "could not change schema dependency for relation \"%s\"", - NameStr(classForm->relname)); + if (hasDependEntry) + { + LockNotPinnedObject(NamespaceRelationId, newNspOid); + if (changeDependencyFor(RelationRelationId, + relOid, + NamespaceRelationId, + oldNspOid, + newNspOid) != 1) + elog(ERROR, "could not change schema dependency for relation \"%s\"", + NameStr(classForm->relname)); + } } if (!already_done) { diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c index 95de402fa6..44b15349b3 100644 --- a/src/backend/commands/trigger.c +++ b/src/backend/commands/trigger.c @@ -1018,8 +1018,6 @@ CreateTriggerFiringOn(CreateTrigStmt *stmt, const char *queryString, ((Form_pg_class) GETSTRUCT(tuple))->relhastriggers = true; CatalogTupleUpdate(pgrel, &tuple->t_self, tuple); - - CommandCounterIncrement(); } else CacheInvalidateRelcacheByTuple(tuple); @@ -1027,6 +1025,12 @@ CreateTriggerFiringOn(CreateTrigStmt *stmt, const char *queryString, heap_freetuple(tuple); table_close(pgrel, RowExclusiveLock); + /* + * CommandCounterIncrement here to ensure the new trigger entry is visible + * when we'll check of object existence when recording the dependencies. + */ + CommandCounterIncrement(); + /* * If we're replacing a trigger, flush all the old dependencies before * recording new ones. @@ -1045,6 +1049,7 @@ CreateTriggerFiringOn(CreateTrigStmt *stmt, const char *queryString, referenced.classId = ProcedureRelationId; referenced.objectId = funcoid; referenced.objectSubId = 0; + LockNotPinnedObject(ProcedureRelationId, funcoid); recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL); if (isInternal && OidIsValid(constraintOid)) @@ -1058,6 +1063,7 @@ CreateTriggerFiringOn(CreateTrigStmt *stmt, const char *queryString, referenced.classId = ConstraintRelationId; referenced.objectId = constraintOid; referenced.objectSubId = 0; + LockNotPinnedObject(ConstraintRelationId, constraintOid); recordDependencyOn(&myself, &referenced, DEPENDENCY_INTERNAL); } else @@ -1070,6 +1076,7 @@ CreateTriggerFiringOn(CreateTrigStmt *stmt, const char *queryString, referenced.classId = RelationRelationId; referenced.objectId = RelationGetRelid(rel); referenced.objectSubId = 0; + /* XXX Do we need a lock for RelationRelationId? */ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO); if (OidIsValid(constrrelid)) @@ -1077,6 +1084,7 @@ CreateTriggerFiringOn(CreateTrigStmt *stmt, const char *queryString, referenced.classId = RelationRelationId; referenced.objectId = constrrelid; referenced.objectSubId = 0; + /* XXX Do we need a lock for RelationRelationId? */ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO); } /* Not possible to have an index dependency in this case */ @@ -1091,6 +1099,7 @@ CreateTriggerFiringOn(CreateTrigStmt *stmt, const char *queryString, referenced.classId = ConstraintRelationId; referenced.objectId = constraintOid; referenced.objectSubId = 0; + LockNotPinnedObject(TriggerRelationId, trigoid); recordDependencyOn(&referenced, &myself, DEPENDENCY_INTERNAL); } @@ -1100,8 +1109,10 @@ CreateTriggerFiringOn(CreateTrigStmt *stmt, const char *queryString, if (OidIsValid(parentTriggerOid)) { ObjectAddressSet(referenced, TriggerRelationId, parentTriggerOid); + LockNotPinnedObject(TriggerRelationId, parentTriggerOid); recordDependencyOn(&myself, &referenced, DEPENDENCY_PARTITION_PRI); ObjectAddressSet(referenced, RelationRelationId, RelationGetRelid(rel)); + /* XXX Do we need a lock for RelationRelationId? */ recordDependencyOn(&myself, &referenced, DEPENDENCY_PARTITION_SEC); } } @@ -1116,6 +1127,7 @@ CreateTriggerFiringOn(CreateTrigStmt *stmt, const char *queryString, for (i = 0; i < ncolumns; i++) { referenced.objectSubId = columns[i]; + /* XXX Do we need a lock for RelationRelationId? */ recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL); } } @@ -1255,9 +1267,11 @@ TriggerSetParentTrigger(Relation trigRel, ObjectAddressSet(depender, TriggerRelationId, childTrigId); ObjectAddressSet(referenced, TriggerRelationId, parentTrigId); + LockNotPinnedObject(TriggerRelationId, parentTrigId); recordDependencyOn(&depender, &referenced, DEPENDENCY_PARTITION_PRI); ObjectAddressSet(referenced, RelationRelationId, childTableId); + /* XXX Do we need a lock for RelationRelationId? */ recordDependencyOn(&depender, &referenced, DEPENDENCY_PARTITION_SEC); } else diff --git a/src/backend/commands/tsearchcmds.c b/src/backend/commands/tsearchcmds.c index b7b5019f1e..29e02f4946 100644 --- a/src/backend/commands/tsearchcmds.c +++ b/src/backend/commands/tsearchcmds.c @@ -66,12 +66,12 @@ static DefElem *buildDefItem(const char *name, const char *val, /* --------------------- TS Parser commands ------------------------ */ /* - * lookup a parser support function and return its OID (as a Datum) + * lookup a parser support function and return its OID * * attnum is the pg_ts_parser column the function will go into */ -static Datum -get_ts_parser_func(DefElem *defel, int attnum) +static Oid +get_ts_parser_func_oid(DefElem *defel, int attnum) { List *funcName = defGetQualifiedName(defel); Oid typeId[3]; @@ -125,7 +125,7 @@ get_ts_parser_func(DefElem *defel, int attnum) func_signature_string(funcName, nargs, NIL, typeId), format_type_be(retTypeId)))); - return ObjectIdGetDatum(procOid); + return procOid; } /* @@ -214,6 +214,7 @@ DefineTSParser(List *names, List *parameters) namestrcpy(&pname, prsname); values[Anum_pg_ts_parser_prsname - 1] = NameGetDatum(&pname); values[Anum_pg_ts_parser_prsnamespace - 1] = ObjectIdGetDatum(namespaceoid); + LockNotPinnedObject(NamespaceRelationId, namespaceoid); /* * loop over the definition list and extract the information we need. @@ -224,28 +225,38 @@ DefineTSParser(List *names, List *parameters) if (strcmp(defel->defname, "start") == 0) { - values[Anum_pg_ts_parser_prsstart - 1] = - get_ts_parser_func(defel, Anum_pg_ts_parser_prsstart); + Oid procoid = get_ts_parser_func_oid(defel, Anum_pg_ts_parser_prsstart); + + values[Anum_pg_ts_parser_prsstart - 1] = ObjectIdGetDatum(procoid); + LockNotPinnedObject(ProcedureRelationId, procoid); } else if (strcmp(defel->defname, "gettoken") == 0) { - values[Anum_pg_ts_parser_prstoken - 1] = - get_ts_parser_func(defel, Anum_pg_ts_parser_prstoken); + Oid procoid = get_ts_parser_func_oid(defel, Anum_pg_ts_parser_prstoken); + + values[Anum_pg_ts_parser_prstoken - 1] = ObjectIdGetDatum(procoid); + LockNotPinnedObject(ProcedureRelationId, procoid); } else if (strcmp(defel->defname, "end") == 0) { - values[Anum_pg_ts_parser_prsend - 1] = - get_ts_parser_func(defel, Anum_pg_ts_parser_prsend); + Oid procoid = get_ts_parser_func_oid(defel, Anum_pg_ts_parser_prsend); + + values[Anum_pg_ts_parser_prsend - 1] = ObjectIdGetDatum(procoid); + LockNotPinnedObject(ProcedureRelationId, procoid); } else if (strcmp(defel->defname, "headline") == 0) { - values[Anum_pg_ts_parser_prsheadline - 1] = - get_ts_parser_func(defel, Anum_pg_ts_parser_prsheadline); + Oid procoid = get_ts_parser_func_oid(defel, Anum_pg_ts_parser_prsheadline); + + values[Anum_pg_ts_parser_prsheadline - 1] = ObjectIdGetDatum(procoid); + LockNotPinnedObject(ProcedureRelationId, procoid); } else if (strcmp(defel->defname, "lextypes") == 0) { - values[Anum_pg_ts_parser_prslextype - 1] = - get_ts_parser_func(defel, Anum_pg_ts_parser_prslextype); + Oid procoid = get_ts_parser_func_oid(defel, Anum_pg_ts_parser_prslextype); + + values[Anum_pg_ts_parser_prslextype - 1] = ObjectIdGetDatum(procoid); + LockNotPinnedObject(ProcedureRelationId, procoid); } else ereport(ERROR, @@ -474,6 +485,10 @@ DefineTSDictionary(List *names, List *parameters) CatalogTupleInsert(dictRel, tup); + /* Lock objects */ + LockNotPinnedObject(NamespaceRelationId, namespaceoid); + LockNotPinnedObject(TSTemplateRelationId, templId); + address = makeDictionaryDependencies(tup); /* Post creation hook for new text search dictionary */ @@ -601,12 +616,12 @@ AlterTSDictionary(AlterTSDictionaryStmt *stmt) /* ---------------------- TS Template commands -----------------------*/ /* - * lookup a template support function and return its OID (as a Datum) + * lookup a template support function and return its OID * * attnum is the pg_ts_template column the function will go into */ -static Datum -get_ts_template_func(DefElem *defel, int attnum) +static Oid +get_ts_template_func_oid(DefElem *defel, int attnum) { List *funcName = defGetQualifiedName(defel); Oid typeId[4]; @@ -642,7 +657,7 @@ get_ts_template_func(DefElem *defel, int attnum) func_signature_string(funcName, nargs, NIL, typeId), format_type_be(retTypeId)))); - return ObjectIdGetDatum(procOid); + return procOid; } /* @@ -723,6 +738,7 @@ DefineTSTemplate(List *names, List *parameters) namestrcpy(&dname, tmplname); values[Anum_pg_ts_template_tmplname - 1] = NameGetDatum(&dname); values[Anum_pg_ts_template_tmplnamespace - 1] = ObjectIdGetDatum(namespaceoid); + LockNotPinnedObject(NamespaceRelationId, namespaceoid); /* * loop over the definition list and extract the information we need. @@ -733,15 +749,19 @@ DefineTSTemplate(List *names, List *parameters) if (strcmp(defel->defname, "init") == 0) { - values[Anum_pg_ts_template_tmplinit - 1] = - get_ts_template_func(defel, Anum_pg_ts_template_tmplinit); + Oid procoid = get_ts_template_func_oid(defel, Anum_pg_ts_template_tmplinit); + + values[Anum_pg_ts_template_tmplinit - 1] = ObjectIdGetDatum(procoid); nulls[Anum_pg_ts_template_tmplinit - 1] = false; + LockNotPinnedObject(ProcedureRelationId, procoid); } else if (strcmp(defel->defname, "lexize") == 0) { - values[Anum_pg_ts_template_tmpllexize - 1] = - get_ts_template_func(defel, Anum_pg_ts_template_tmpllexize); + Oid procoid = get_ts_template_func_oid(defel, Anum_pg_ts_template_tmpllexize); + + values[Anum_pg_ts_template_tmpllexize - 1] = ObjectIdGetDatum(procoid); nulls[Anum_pg_ts_template_tmpllexize - 1] = false; + LockNotPinnedObject(ProcedureRelationId, procoid); } else ereport(ERROR, @@ -879,6 +899,7 @@ makeConfigurationDependencies(HeapTuple tuple, bool removeOld, referenced.objectId = cfgmap->mapdict; referenced.objectSubId = 0; add_exact_object_address(&referenced, addrs); + LockNotPinnedObject(TSDictionaryRelationId, cfgmap->mapdict); } systable_endscan(scan); @@ -998,6 +1019,10 @@ DefineTSConfiguration(List *names, List *parameters, ObjectAddress *copied) values[Anum_pg_ts_config_cfgowner - 1] = ObjectIdGetDatum(GetUserId()); values[Anum_pg_ts_config_cfgparser - 1] = ObjectIdGetDatum(prsOid); + /* Lock objects */ + LockNotPinnedObject(NamespaceRelationId, namespaceoid); + LockNotPinnedObject(TSParserRelationId, prsOid); + tup = heap_form_tuple(cfgRel->rd_att, values, nulls); CatalogTupleInsert(cfgRel, tup); @@ -1156,6 +1181,7 @@ ObjectAddress AlterTSConfiguration(AlterTSConfigurationStmt *stmt) { HeapTuple tup; + Form_pg_ts_config cfg; Oid cfgId; Relation relMap; ObjectAddress address; @@ -1168,7 +1194,8 @@ AlterTSConfiguration(AlterTSConfigurationStmt *stmt) errmsg("text search configuration \"%s\" does not exist", NameListToString(stmt->cfgname)))); - cfgId = ((Form_pg_ts_config) GETSTRUCT(tup))->oid; + cfg = (Form_pg_ts_config) GETSTRUCT(tup); + cfgId = cfg->oid; /* must be owner */ if (!object_ownercheck(TSConfigRelationId, cfgId, GetUserId())) @@ -1183,6 +1210,10 @@ AlterTSConfiguration(AlterTSConfigurationStmt *stmt) else if (stmt->tokentype) DropConfigurationMapping(stmt, tup, relMap); + /* Lock dependent objects */ + LockNotPinnedObject(NamespaceRelationId, cfg->cfgnamespace); + LockNotPinnedObject(TSParserRelationId, cfg->cfgparser); + /* Update dependencies */ makeConfigurationDependencies(tup, true, relMap); @@ -1414,6 +1445,8 @@ MakeConfigurationMapping(AlterTSConfigurationStmt *stmt, repl_val[Anum_pg_ts_config_map_mapdict - 1] = ObjectIdGetDatum(dictNew); repl_repl[Anum_pg_ts_config_map_mapdict - 1] = true; + LockNotPinnedObject(TSDictionaryRelationId, dictNew); + newtup = heap_modify_tuple(maptup, RelationGetDescr(relMap), repl_val, repl_null, repl_repl); @@ -1456,6 +1489,8 @@ MakeConfigurationMapping(AlterTSConfigurationStmt *stmt, slot[slotCount]->tts_values[Anum_pg_ts_config_map_mapseqno - 1] = Int32GetDatum(j + 1); slot[slotCount]->tts_values[Anum_pg_ts_config_map_mapdict - 1] = ObjectIdGetDatum(dictIds[j]); + LockNotPinnedObject(TSDictionaryRelationId, dictIds[j]); + ExecStoreVirtualTuple(slot[slotCount]); slotCount++; diff --git a/src/backend/commands/typecmds.c b/src/backend/commands/typecmds.c index 2a1e713335..9febaa24a7 100644 --- a/src/backend/commands/typecmds.c +++ b/src/backend/commands/typecmds.c @@ -1794,6 +1794,7 @@ makeRangeConstructors(const char *name, Oid namespace, * that they go away silently when the type is dropped. Note that * pg_dump depends on this choice to avoid dumping the constructors. */ + LockNotPinnedObject(TypeRelationId, rangeOid); recordDependencyOn(&myself, &referenced, DEPENDENCY_INTERNAL); } } @@ -1859,6 +1860,7 @@ makeMultirangeConstructors(const char *name, Oid namespace, * that they go away silently when the type is dropped. Note that pg_dump * depends on this choice to avoid dumping the constructors. */ + LockNotPinnedObject(TypeRelationId, multirangeOid); recordDependencyOn(&myself, &referenced, DEPENDENCY_INTERNAL); pfree(argtypes); @@ -2672,6 +2674,45 @@ AlterDomainDefault(List *names, Node *defaultRaw) CatalogTupleUpdate(rel, &tup->t_self, newtuple); + /* Lock dependent objects */ + typTup = (Form_pg_type) GETSTRUCT(newtuple); + + if (OidIsValid(typTup->typnamespace)) + LockNotPinnedObject(NamespaceRelationId, typTup->typnamespace); + + if (OidIsValid(typTup->typinput)) + LockNotPinnedObject(ProcedureRelationId, typTup->typinput); + + if (OidIsValid(typTup->typoutput)) + LockNotPinnedObject(ProcedureRelationId, typTup->typoutput); + + if (OidIsValid(typTup->typreceive)) + LockNotPinnedObject(ProcedureRelationId, typTup->typreceive); + + if (OidIsValid(typTup->typsend)) + LockNotPinnedObject(ProcedureRelationId, typTup->typsend); + + if (OidIsValid(typTup->typmodin)) + LockNotPinnedObject(ProcedureRelationId, typTup->typmodin); + + if (OidIsValid(typTup->typmodout)) + LockNotPinnedObject(ProcedureRelationId, typTup->typmodout); + + if (OidIsValid(typTup->typanalyze)) + LockNotPinnedObject(ProcedureRelationId, typTup->typanalyze); + + if (OidIsValid(typTup->typsubscript)) + LockNotPinnedObject(ProcedureRelationId, typTup->typsubscript); + + if (OidIsValid(typTup->typbasetype)) + LockNotPinnedObject(TypeRelationId, typTup->typbasetype); + + if (OidIsValid(typTup->typcollation)) + LockNotPinnedObject(CollationRelationId, typTup->typcollation); + + if (OidIsValid(typTup->typelem)) + LockNotPinnedObject(TypeRelationId, typTup->typelem); + /* Rebuild dependencies */ GenerateTypeDependencies(newtuple, rel, @@ -4276,10 +4317,13 @@ AlterTypeNamespaceInternal(Oid typeOid, Oid nspOid, if (oldNspOid != nspOid && (isCompositeType || typform->typtype != TYPTYPE_COMPOSITE) && !isImplicitArray) + { + LockNotPinnedObject(NamespaceRelationId, nspOid); if (changeDependencyFor(TypeRelationId, typeOid, NamespaceRelationId, oldNspOid, nspOid) != 1) elog(ERROR, "could not change schema dependency for type \"%s\"", format_type_be(typeOid)); + } InvokeObjectPostAlterHook(TypeRelationId, typeOid, 0); @@ -4571,6 +4615,7 @@ AlterTypeRecurse(Oid typeOid, bool isImplicitArray, SysScanDesc scan; ScanKeyData key[1]; HeapTuple domainTup; + Form_pg_type typeForm; /* Since this function recurses, it could be driven to stack overflow */ check_stack_depth(); @@ -4619,6 +4664,45 @@ AlterTypeRecurse(Oid typeOid, bool isImplicitArray, newtup = heap_modify_tuple(tup, RelationGetDescr(catalog), values, nulls, replaces); + /* Lock dependent objects */ + typeForm = (Form_pg_type) GETSTRUCT(newtup); + + if (OidIsValid(typeForm->typnamespace)) + LockNotPinnedObject(NamespaceRelationId, typeForm->typnamespace); + + if (OidIsValid(typeForm->typinput)) + LockNotPinnedObject(ProcedureRelationId, typeForm->typinput); + + if (OidIsValid(typeForm->typoutput)) + LockNotPinnedObject(ProcedureRelationId, typeForm->typoutput); + + if (OidIsValid(typeForm->typreceive)) + LockNotPinnedObject(ProcedureRelationId, typeForm->typreceive); + + if (OidIsValid(typeForm->typsend)) + LockNotPinnedObject(ProcedureRelationId, typeForm->typsend); + + if (OidIsValid(typeForm->typmodin)) + LockNotPinnedObject(ProcedureRelationId, typeForm->typmodin); + + if (OidIsValid(typeForm->typmodout)) + LockNotPinnedObject(ProcedureRelationId, typeForm->typmodout); + + if (OidIsValid(typeForm->typanalyze)) + LockNotPinnedObject(ProcedureRelationId, typeForm->typanalyze); + + if (OidIsValid(typeForm->typsubscript)) + LockNotPinnedObject(ProcedureRelationId, typeForm->typsubscript); + + if (OidIsValid(typeForm->typbasetype)) + LockNotPinnedObject(TypeRelationId, typeForm->typbasetype); + + if (OidIsValid(typeForm->typcollation)) + LockNotPinnedObject(CollationRelationId, typeForm->typcollation); + + if (OidIsValid(typeForm->typelem)) + LockNotPinnedObject(TypeRelationId, typeForm->typelem); + CatalogTupleUpdate(catalog, &newtup->t_self, newtup); /* Rebuild dependencies for this type */ diff --git a/src/backend/rewrite/rewriteDefine.c b/src/backend/rewrite/rewriteDefine.c index 6cc9a8d8bf..ccd03ceeb8 100644 --- a/src/backend/rewrite/rewriteDefine.c +++ b/src/backend/rewrite/rewriteDefine.c @@ -155,6 +155,7 @@ InsertRule(const char *rulname, referenced.objectId = eventrel_oid; referenced.objectSubId = 0; + /* XXX Do we need a lock for RelationRelationId? */ recordDependencyOn(&myself, &referenced, (evtype == CMD_SELECT) ? DEPENDENCY_INTERNAL : DEPENDENCY_AUTO); diff --git a/src/backend/utils/errcodes.txt b/src/backend/utils/errcodes.txt index 3250d539e1..60e8539fe3 100644 --- a/src/backend/utils/errcodes.txt +++ b/src/backend/utils/errcodes.txt @@ -271,6 +271,7 @@ Section: Class 28 - Invalid Authorization Specification Section: Class 2B - Dependent Privilege Descriptors Still Exist 2B000 E ERRCODE_DEPENDENT_PRIVILEGE_DESCRIPTORS_STILL_EXIST dependent_privilege_descriptors_still_exist +2BP02 E ERRCODE_DEPENDENT_OBJECTS_DOES_NOT_EXIST dependent_objects_does_not_exist 2BP01 E ERRCODE_DEPENDENT_OBJECTS_STILL_EXIST dependent_objects_still_exist Section: Class 2D - Invalid Transaction Termination diff --git a/src/include/catalog/dependency.h b/src/include/catalog/dependency.h index 7eee66f810..c57204cc40 100644 --- a/src/include/catalog/dependency.h +++ b/src/include/catalog/dependency.h @@ -101,6 +101,9 @@ typedef struct ObjectAddresses ObjectAddresses; /* in dependency.c */ extern void AcquireDeletionLock(const ObjectAddress *object, int flags); +extern void LockNotPinnedObjectById(const ObjectAddress *object); +extern void LockNotPinnedObjectsById(const ObjectAddress *object, int nobject); +extern void LockNotPinnedObject(Oid classid, Oid objid); extern void ReleaseDeletionLock(const ObjectAddress *object); @@ -128,6 +131,9 @@ extern void add_exact_object_address(const ObjectAddress *object, extern bool object_address_present(const ObjectAddress *object, const ObjectAddresses *addrs); +extern void lock_record_object_address_dependencies(const ObjectAddress *depender, + ObjectAddresses *referenced, + DependencyType behavior); extern void record_object_address_dependencies(const ObjectAddress *depender, ObjectAddresses *referenced, DependencyType behavior); @@ -172,6 +178,7 @@ extern long changeDependenciesOf(Oid classId, Oid oldObjectId, extern long changeDependenciesOn(Oid refClassId, Oid oldRefObjectId, Oid newRefObjectId); +extern bool isObjectPinned(const ObjectAddress *object); extern Oid getExtensionOfObject(Oid classId, Oid objectId); extern List *getAutoExtensionsOfObject(Oid classId, Oid objectId); diff --git a/src/include/catalog/objectaddress.h b/src/include/catalog/objectaddress.h index 3a70d80e32..56f746264b 100644 --- a/src/include/catalog/objectaddress.h +++ b/src/include/catalog/objectaddress.h @@ -53,6 +53,7 @@ extern void check_object_ownership(Oid roleid, Node *object, Relation relation); extern Oid get_object_namespace(const ObjectAddress *address); +extern bool ObjectByIdExist(const ObjectAddress *address); extern bool is_objectclass_supported(Oid class_id); extern const char *get_object_class_descr(Oid class_id); diff --git a/src/test/isolation/expected/test_dependencies_locks.out b/src/test/isolation/expected/test_dependencies_locks.out new file mode 100644 index 0000000000..9b645d7aa5 --- /dev/null +++ b/src/test/isolation/expected/test_dependencies_locks.out @@ -0,0 +1,129 @@ +Parsed test spec with 2 sessions + +starting permutation: s1_begin s1_create_function_in_schema s2_drop_schema s1_commit +step s1_begin: BEGIN; +step s1_create_function_in_schema: CREATE FUNCTION testschema.foo() RETURNS int AS 'select 1' LANGUAGE sql; +step s2_drop_schema: DROP SCHEMA testschema; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_schema: <... completed> +ERROR: cannot drop schema testschema because other objects depend on it + +starting permutation: s2_begin s2_drop_schema s1_create_function_in_schema s2_commit +step s2_begin: BEGIN; +step s2_drop_schema: DROP SCHEMA testschema; +step s1_create_function_in_schema: CREATE FUNCTION testschema.foo() RETURNS int AS 'select 1' LANGUAGE sql; <waiting ...> +step s2_commit: COMMIT; +step s1_create_function_in_schema: <... completed> +ERROR: schema testschema does not exist + +starting permutation: s1_begin s1_alter_function_schema s2_drop_alterschema s1_commit +step s1_begin: BEGIN; +step s1_alter_function_schema: ALTER FUNCTION public.falter() SET SCHEMA alterschema; +step s2_drop_alterschema: DROP SCHEMA alterschema; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_alterschema: <... completed> +ERROR: cannot drop schema alterschema because other objects depend on it + +starting permutation: s2_begin s2_drop_alterschema s1_alter_function_schema s2_commit +step s2_begin: BEGIN; +step s2_drop_alterschema: DROP SCHEMA alterschema; +step s1_alter_function_schema: ALTER FUNCTION public.falter() SET SCHEMA alterschema; <waiting ...> +step s2_commit: COMMIT; +step s1_alter_function_schema: <... completed> +ERROR: schema alterschema does not exist + +starting permutation: s1_begin s1_create_function_with_argtype s2_drop_foo_type s1_commit +step s1_begin: BEGIN; +step s1_create_function_with_argtype: CREATE FUNCTION fooargtype(num foo) RETURNS int AS 'select 1' LANGUAGE sql; +step s2_drop_foo_type: DROP TYPE public.foo; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_foo_type: <... completed> +ERROR: cannot drop type foo because other objects depend on it + +starting permutation: s2_begin s2_drop_foo_type s1_create_function_with_argtype s2_commit +step s2_begin: BEGIN; +step s2_drop_foo_type: DROP TYPE public.foo; +step s1_create_function_with_argtype: CREATE FUNCTION fooargtype(num foo) RETURNS int AS 'select 1' LANGUAGE sql; <waiting ...> +step s2_commit: COMMIT; +step s1_create_function_with_argtype: <... completed> +ERROR: type foo does not exist + +starting permutation: s1_begin s1_create_function_with_rettype s2_drop_foo_rettype s1_commit +step s1_begin: BEGIN; +step s1_create_function_with_rettype: CREATE FUNCTION footrettype() RETURNS id LANGUAGE sql RETURN 1; +step s2_drop_foo_rettype: DROP DOMAIN id; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_foo_rettype: <... completed> +ERROR: cannot drop type id because other objects depend on it + +starting permutation: s2_begin s2_drop_foo_rettype s1_create_function_with_rettype s2_commit +step s2_begin: BEGIN; +step s2_drop_foo_rettype: DROP DOMAIN id; +step s1_create_function_with_rettype: CREATE FUNCTION footrettype() RETURNS id LANGUAGE sql RETURN 1; <waiting ...> +step s2_commit: COMMIT; +step s1_create_function_with_rettype: <... completed> +ERROR: type id does not exist + +starting permutation: s1_begin s1_create_function_with_function s2_drop_function_f s1_commit +step s1_begin: BEGIN; +step s1_create_function_with_function: CREATE FUNCTION foofunc() RETURNS int LANGUAGE SQL RETURN f() + 1; +step s2_drop_function_f: DROP FUNCTION f(); <waiting ...> +step s1_commit: COMMIT; +step s2_drop_function_f: <... completed> +ERROR: cannot drop function f() because other objects depend on it + +starting permutation: s2_begin s2_drop_function_f s1_create_function_with_function s2_commit +step s2_begin: BEGIN; +step s2_drop_function_f: DROP FUNCTION f(); +step s1_create_function_with_function: CREATE FUNCTION foofunc() RETURNS int LANGUAGE SQL RETURN f() + 1; <waiting ...> +step s2_commit: COMMIT; +step s1_create_function_with_function: <... completed> +ERROR: function f() does not exist + +starting permutation: s1_begin s1_create_domain_with_domain s2_drop_domain_id s1_commit +step s1_begin: BEGIN; +step s1_create_domain_with_domain: CREATE DOMAIN idid as id; +step s2_drop_domain_id: DROP DOMAIN id; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_domain_id: <... completed> +ERROR: cannot drop type id because other objects depend on it + +starting permutation: s2_begin s2_drop_domain_id s1_create_domain_with_domain s2_commit +step s2_begin: BEGIN; +step s2_drop_domain_id: DROP DOMAIN id; +step s1_create_domain_with_domain: CREATE DOMAIN idid as id; <waiting ...> +step s2_commit: COMMIT; +step s1_create_domain_with_domain: <... completed> +ERROR: type id does not exist + +starting permutation: s1_begin s1_create_table_with_type s2_drop_footab_type s1_commit +step s1_begin: BEGIN; +step s1_create_table_with_type: CREATE TABLE tabtype(a footab); +step s2_drop_footab_type: DROP TYPE public.footab; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_footab_type: <... completed> +ERROR: cannot drop type footab because other objects depend on it + +starting permutation: s2_begin s2_drop_footab_type s1_create_table_with_type s2_commit +step s2_begin: BEGIN; +step s2_drop_footab_type: DROP TYPE public.footab; +step s1_create_table_with_type: CREATE TABLE tabtype(a footab); <waiting ...> +step s2_commit: COMMIT; +step s1_create_table_with_type: <... completed> +ERROR: type footab does not exist + +starting permutation: s1_begin s1_create_server_with_fdw_wrapper s2_drop_fdw_wrapper s1_commit +step s1_begin: BEGIN; +step s1_create_server_with_fdw_wrapper: CREATE SERVER srv_fdw_wrapper FOREIGN DATA WRAPPER fdw_wrapper; +step s2_drop_fdw_wrapper: DROP FOREIGN DATA WRAPPER fdw_wrapper RESTRICT; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_fdw_wrapper: <... completed> +ERROR: cannot drop foreign-data wrapper fdw_wrapper because other objects depend on it + +starting permutation: s2_begin s2_drop_fdw_wrapper s1_create_server_with_fdw_wrapper s2_commit +step s2_begin: BEGIN; +step s2_drop_fdw_wrapper: DROP FOREIGN DATA WRAPPER fdw_wrapper RESTRICT; +step s1_create_server_with_fdw_wrapper: CREATE SERVER srv_fdw_wrapper FOREIGN DATA WRAPPER fdw_wrapper; <waiting ...> +step s2_commit: COMMIT; +step s1_create_server_with_fdw_wrapper: <... completed> +ERROR: foreign-data wrapper fdw_wrapper does not exist diff --git a/src/test/isolation/isolation_schedule b/src/test/isolation/isolation_schedule index 0342eb39e4..1b67f0bffe 100644 --- a/src/test/isolation/isolation_schedule +++ b/src/test/isolation/isolation_schedule @@ -114,3 +114,4 @@ test: serializable-parallel-2 test: serializable-parallel-3 test: matview-write-skew test: lock-nowait +test: test_dependencies_locks diff --git a/src/test/isolation/specs/test_dependencies_locks.spec b/src/test/isolation/specs/test_dependencies_locks.spec new file mode 100644 index 0000000000..5d04dfe9dc --- /dev/null +++ b/src/test/isolation/specs/test_dependencies_locks.spec @@ -0,0 +1,89 @@ +setup +{ + CREATE SCHEMA testschema; + CREATE SCHEMA alterschema; + CREATE TYPE public.foo as enum ('one', 'two'); + CREATE TYPE public.footab as enum ('three', 'four'); + CREATE DOMAIN id AS int; + CREATE FUNCTION f() RETURNS int LANGUAGE SQL RETURN 1; + CREATE FUNCTION public.falter() RETURNS int LANGUAGE SQL RETURN 1; + CREATE FOREIGN DATA WRAPPER fdw_wrapper; +} + +teardown +{ + DROP FUNCTION IF EXISTS testschema.foo(); + DROP FUNCTION IF EXISTS fooargtype(num foo); + DROP FUNCTION IF EXISTS footrettype(); + DROP FUNCTION IF EXISTS foofunc(); + DROP FUNCTION IF EXISTS public.falter(); + DROP FUNCTION IF EXISTS alterschema.falter(); + DROP DOMAIN IF EXISTS idid; + DROP SERVER IF EXISTS srv_fdw_wrapper; + DROP TABLE IF EXISTS tabtype; + DROP SCHEMA IF EXISTS testschema; + DROP SCHEMA IF EXISTS alterschema; + DROP TYPE IF EXISTS public.foo; + DROP TYPE IF EXISTS public.footab; + DROP DOMAIN IF EXISTS id; + DROP FUNCTION IF EXISTS f(); + DROP FOREIGN DATA WRAPPER IF EXISTS fdw_wrapper; +} + +session "s1" + +step "s1_begin" { BEGIN; } +step "s1_create_function_in_schema" { CREATE FUNCTION testschema.foo() RETURNS int AS 'select 1' LANGUAGE sql; } +step "s1_create_function_with_argtype" { CREATE FUNCTION fooargtype(num foo) RETURNS int AS 'select 1' LANGUAGE sql; } +step "s1_create_function_with_rettype" { CREATE FUNCTION footrettype() RETURNS id LANGUAGE sql RETURN 1; } +step "s1_create_function_with_function" { CREATE FUNCTION foofunc() RETURNS int LANGUAGE SQL RETURN f() + 1; } +step "s1_alter_function_schema" { ALTER FUNCTION public.falter() SET SCHEMA alterschema; } +step "s1_create_domain_with_domain" { CREATE DOMAIN idid as id; } +step "s1_create_table_with_type" { CREATE TABLE tabtype(a footab); } +step "s1_create_server_with_fdw_wrapper" { CREATE SERVER srv_fdw_wrapper FOREIGN DATA WRAPPER fdw_wrapper; } +step "s1_commit" { COMMIT; } + +session "s2" + +step "s2_begin" { BEGIN; } +step "s2_drop_schema" { DROP SCHEMA testschema; } +step "s2_drop_alterschema" { DROP SCHEMA alterschema; } +step "s2_drop_foo_type" { DROP TYPE public.foo; } +step "s2_drop_foo_rettype" { DROP DOMAIN id; } +step "s2_drop_footab_type" { DROP TYPE public.footab; } +step "s2_drop_function_f" { DROP FUNCTION f(); } +step "s2_drop_domain_id" { DROP DOMAIN id; } +step "s2_drop_fdw_wrapper" { DROP FOREIGN DATA WRAPPER fdw_wrapper RESTRICT; } +step "s2_commit" { COMMIT; } + +# function - schema +permutation "s1_begin" "s1_create_function_in_schema" "s2_drop_schema" "s1_commit" +permutation "s2_begin" "s2_drop_schema" "s1_create_function_in_schema" "s2_commit" + +# alter function - schema +permutation "s1_begin" "s1_alter_function_schema" "s2_drop_alterschema" "s1_commit" +permutation "s2_begin" "s2_drop_alterschema" "s1_alter_function_schema" "s2_commit" + +# function - argtype +permutation "s1_begin" "s1_create_function_with_argtype" "s2_drop_foo_type" "s1_commit" +permutation "s2_begin" "s2_drop_foo_type" "s1_create_function_with_argtype" "s2_commit" + +# function - rettype +permutation "s1_begin" "s1_create_function_with_rettype" "s2_drop_foo_rettype" "s1_commit" +permutation "s2_begin" "s2_drop_foo_rettype" "s1_create_function_with_rettype" "s2_commit" + +# function - function +permutation "s1_begin" "s1_create_function_with_function" "s2_drop_function_f" "s1_commit" +permutation "s2_begin" "s2_drop_function_f" "s1_create_function_with_function" "s2_commit" + +# domain - domain +permutation "s1_begin" "s1_create_domain_with_domain" "s2_drop_domain_id" "s1_commit" +permutation "s2_begin" "s2_drop_domain_id" "s1_create_domain_with_domain" "s2_commit" + +# table - type +permutation "s1_begin" "s1_create_table_with_type" "s2_drop_footab_type" "s1_commit" +permutation "s2_begin" "s2_drop_footab_type" "s1_create_table_with_type" "s2_commit" + +# server - foreign data wrapper +permutation "s1_begin" "s1_create_server_with_fdw_wrapper" "s2_drop_fdw_wrapper" "s1_commit" +permutation "s2_begin" "s2_drop_fdw_wrapper" "s1_create_server_with_fdw_wrapper" "s2_commit" -- 2.34.1 --bmyJBR+1NCLqhmdA-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v52 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gp2pyozrd5pweboh Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v52-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v47 7/9] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0008-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v48 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v49 4/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b86ad73c626..bac8d527ae5 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4a2d21915b1..25e8c1945c5 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --3n3vqr5y2jhknrqj Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v49-0005-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v50 4/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --brnevsqjnuzpyok4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v50-0005-support-repacking-tables-with-exclusion-constrai.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v51 04/10] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 38ef683d4c7..60fb7051830 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1183,7 +1183,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..4b8ed2c7660 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,6 +1453,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1538,7 +1549,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3020,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3031,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3083,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3841,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..51b27a8c71c 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index cba379810c7..932924c13e0 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..7ebe4f0bd87 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --r2slln3zpmilwu22 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v51-0005-invert-meaning-of-index_create-flag-bit.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v53 2/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 26 +++++++++++++++++++++----- src/backend/catalog/toasting.c | 3 ++- src/include/catalog/index.h | 4 +++- 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index c52c0a6023d..ebd41176b94 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1184,7 +1184,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index e418d67e8e4..36712b1bcc2 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_SUPPRESS_PROGRESS: + * don't report progress during the index build. + * * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_SUPPRESS_PROGRESS) == 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1448,8 +1453,16 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * The current callers do not need to report progress: if 'concurrently' + * is true, there is no build at all to report about; and otherwise the + * index build is a sub-command of REPACK, and the current progress + * reporting infrastructure does not allow two commands to report their + * progress at the same time. + */ if (concurrently) - flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT | + INDEX_CREATE_SUPPRESS_PROGRESS; /* * Now create the new index. @@ -1538,7 +1551,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3009,6 +3022,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3019,7 +3033,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3070,6 +3085,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3827,7 +3843,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 4aa52a4bd25..dcee536fd3f 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -332,7 +332,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index ed9e4c37d27..9f538dac798 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_SUPPRESS_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --qr3jlalmmcpkiodg Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v53-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v45 6/8] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --cldir6umqisr673d Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v45-0007-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
* [PATCH v46 5/7] Fix a few problems in index build progress reporting. @ 2026-03-27 15:50 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 1275+ messages in thread From: Álvaro Herrera @ 2026-03-27 15:50 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 6 files changed, 27 insertions(+), 8 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --mdsv5bdgfjjj6d77 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v46-0006-Error-out-any-process-that-would-block-at-REPACK.patch" ^ permalink raw reply [nested|flat] 1275+ messages in thread
end of thread, other threads:[~2026-03-27 15:50 UTC | newest] Thread overview: 1275+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2024-03-29 15:43 [PATCH v10] Avoid orphaned objects dependencies Bertrand Drouvot <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v53 2/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v50 4/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v52 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v51 04/10] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v49 4/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v48 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v45 6/8] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v46 5/7] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]> 2026-03-27 15:50 [PATCH v47 7/9] Fix a few problems in index build progress reporting. Álvaro Herrera <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox